Engineering Jul 30, 2026 7 min read
Logs without sampling
Sampling was a storage decision dressed up as a feature. Here is how we index every line and still answer a query over a month of logs in under a second.
Priya Natarajan
Every log product eventually ships a feature called sampling. It is presented as a way to cut cost, and it does that, but the thing it really cuts is the chance that the one line you need survived. The error that happened once, at 03:14:02, to one customer, is precisely the line a one-in-ten sample discards.
Carbon does not sample. This post is about how that is affordable, and what it took.
Columns, not rows
A log line looks like a string, but it is mostly structure: a timestamp, a level, a service, a handful of keys, and a short message. Stored as strings, a month of logs from a busy service is terabytes. Stored as columns, with every field compressed against its neighbours, the same month is a tenth of that, and a query that filters on service and level touches only two of them.
The columnar store is not ours; it is the same one behind most modern analytics warehouses. What is ours is the ingestion path, which parses each line into its fields at the edge, before it is written, so that nothing is stored twice and nothing is scanned that the query did not ask for.
Drop rules, at the edge
Not sampling does not mean keeping everything. Health checks, load-balancer heartbeats, and debug lines from a library you did not write are the bulk of most log volume and the value of almost none of it. A drop rule matches those lines at the edge and discards them before ingestion. The difference from sampling is that you chose what to drop, by looking at it, rather than a random number generator choosing for you.
Every rule shows what it dropped in the last day. Most customers find that three or four rules remove half their volume, and half their bill.
The query path
A query over raw logs is compiled to run against the columnar files directly. There is no index to build and no warm-up: the first query over a new month is as fast as the hundredth. The result in the product screenshot on our home page, 2.5 million matching rows in 700 milliseconds over a month of a service's logs, is a real query from our own account.
What it is not
It is not free. Ingesting everything costs more storage than sampling, and we pass that on per gigabyte. What we will not do is make the choice for you and call it a feature.
More from the blog
- Aug 27, 2026 Why we check every thirty seconds Most monitors check every three to five minutes. That interval was chosen for the monitor's budget, not for yours. Here is what the extra checks actually buy.
- Aug 12, 2026 Writing a status update at 3am The best incident communication we have seen came from people who had a template, a rule about timing, and permission to say 'we do not know yet'.
- Jul 15, 2026 What an SLO is, really Service level objectives get explained with maths. They are easier to understand as a decision about who gets paged, and when.